JEuropaisches Patentamt 
European Patent Office ® Publication number: 0 1 61 512 

AfRno AiirnnMn dfis brevets B1 


® 


® 


Office europeen des brevets 

EUROPEAN PATENT SPECIFICATION 


(§) Dateof publication of patent specification: 19.07.89 (3) Int. Cl.^: H 04 N 17/00, H 04 H 9/00 
(g) Application number: 85104566.6 
(g) Date of filing: 16.04.85 

(D Divisional application 871 18133 filed on 08.12.87. 


(g) Program identification system. 


CM 
U) 

s 


Q. 
UJ 


Pnority: 26.04.84 US 604367 


@ Dateof publication of application: 
21.11.85 Bulletin 85/47 


® Publication of the grant of the patent: 
19.07.89 Bulletin 89/29 


@ Designated Contracting States: 
ATBECH DEFRGBITUNL 


References cited: 
US-A-3 849 729 
US-A-3906450 
US-A-3919479 
US-A-4230 990 

JOURNAL OF THE SMPTE, vol. 84, March 1975, 
pages 162-163, Scarsdale, US; G. AUERBACH: 
''A pattern-recognition system for monitoring 
video and audio signals^ 
ICASSP 83, Boston, 14th-16th April 1983, CH 
1841-6/83/0000-0753, pages 753-756, New 
York, US; P.K. RAJASEKARAN et al.: 
IVIicrocomputer tmplementable low cost 
speaker-Independent word recognition'^ 


® 


® 


® 


Proprietor: A. C. Nielsen Company 
Nielsen Plaza 

Northbrook Illinois 60062 (US) 


Inventor: Kiewit, David A. 
2420 Seneca 
Palm Harbor Rorida (US) 
Inventor: Lu, Daozlieng 
695 San Salvador 
Dunedin Rorida (US) 


Representative: Bauer, Robert Dipl.-lng. et al 
Boeters, Bauer 8l Partner Patentanwalte 
Thomas-Wimmer-Ring 14 
D-8000 Munchen 22 (DE) 


Note: Within nine months from the publication of the mention of the grant of the European patent any person may 
give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall 
be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been 
paid (Art. 99(1 1 European patent convention). 


Courier Press, Leamington Spa, England. 


1 EP 0 

Description 

This invention relates generally to methods and 
apparatus for identifying programs and the view- 
ing habits of the public, and more particularly to a 
system for identifying programs, such as tele- 
vision programs obtained from various sources, 
including video recorders. 

Systems for identifying programs such as tele- 
vision programs that are broadcast or viewed are 
known. Such systems fall into various categories. 
These categories include manual systems 
wherein the viewer keeps a diary indicating all of 
the programs viewed; mechanical, electro- 
mechanical and electronic systems that sense the 
channel to which a television receiver is tuned; 
systems that detect identifying signals present in 
the television broadcast signal; and systems that 
monitor and analyze the program content of the 
broadcast signal utilizing image processing and 
correlation techniques to identify the program. 

While all of these systems do provide a way to 
identify programs and to monitor the viewing 
habits of the public, the manual methods are slow 
in acquiring data and are prone to inaccuracies 
resulting from the entry of erroneous data that 
may be Intentionally or unintentionally entered. 
Systems that monitor the channel to which a 
receiver is tuned overcome the drawbacks of the 
manual systems, but require access to the tuning 
mechanism of the receiver. Such access is 
becoming increasingly difficult with the advent of 
cable and pay television systems that utilize 
various converters and decoders, many of which 
are difficult to access. Moreover, such systems 
cannot identify programs played from a source 
such as a home video recorder. Systems that 
detect identification signals encoded on the 
broadcast signal overcome many of the accuracy 
and access disadvantages described above, but 
require the cooperation of the broadcasters who 
must encode the programs prior to broadcast for 
such systems to be effective. Consequently, such 
systems are useable only on a limited number of 
broadcasts, siich as network broadcasts. Systems 
that monitor and analyze the content of the 
program itself theoretically eliminate the dis- 
advantages of the other systems, but heretofore 
the information processing requirements of such 
systems rendered such systems impractical or 
uneconomical because of the computing capacity 
required to analyze the vast quantity of infor- 
mation present in a broadcast signals. Moreover, 
previous attempts to reduce the quantity of infor- 
mation processed tended to introduce 
inaccuracies. 

An example of a system that utilizes the pro- 
gram content of a broadcast signal to identify the 
program is described In United States Patent No. 
US— A— 3,919,479 to Moon et al. The Moon et al. 
system utilizes a non-linear analog transform to 
produce a low frequency envelope waveform, 
and the information in the low frequency 
envelope of a predetermined time interval is 
digitized to generate a signature. The signatures 
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thus generated are compared with reference sig- 
natures to identify the program. However, the 
Moon et al. system generates the signatures 
continuously, and consequently, requires a large 

5 computer to process the data thus generated. 

Another ' system that monitors the program 
content of a broadcast signal is disclosed in 
United States Patent No. US— A— 4,230,990 to 
Lert Jr. et al. The system disclosed in the Lert Jr. 

w et al patent reduces the amount of data that must 
be processed, as compared to the data amount as 
processed by the Moon et aL patent, by utilizing 
cues that are either externally generated or 
present in the program material to initiate the 

75 signature generation and correlation process only 
after a cue has been detected. However, the 
system disclosed in the Lert Jr. et al. patent is 
designed to monitor the programs broadcast by 
broadcasting stations such as network affiliated 

20 stations in order to determine whether they are 
broadcasting the programs required by the net- 
works to be broadcast, rather than as a system for 
determining the viewing habits of the public. 
Thus, many of the problems that occur when the 

25 habits of a television viewer are being monitored, 
such as, for example, frequent channel changing, 
and the problems that occur in identifying pro- 
grams that are recorded and subsequentiy played 
back by a home video recorder are not addressed. 

30 The invention as claimed solves the problem of 
identifying programs broadcast and the viewing 
habits of television viewers regardless of the 
source of the program material, whether it be off 
the air, cable or prerecorded material, utilizing 

35 data reduction and correlation techniques to mini- 
mize the amount of computation required to 
identify the program material. This is accom- 
plished by extracting signatures from the pro- 
gram material only upon the occurrence of certain 

40 predetermined events, whether the events occur 
in the program material, for example a blank 
frame or a scene change, or are the result of an 
action taken by the viewer, for example the on/off 
condition of the television s6t and the mode of 

45 operation of the video recorder. In addition to 
providing a trigger for the extraction of the signa- 
ture from the program material, the predeter- 
mined sequence of events may serve to indicate 
whether a program Is being recorded or a pre- 

50 recorded program is being viewed. 

In accordance with a preferred embodiment of 
the invention, there is provided a system having a 
home unit that monitors the viewing conditions in 
the home, for example, whether the television set 

55 is on or off, whether any home video recorder 
(e.g., a video cassette recorder or VCR) is on or off 
and the video signal being received. By monitor- 
ing the aforementioned signals and functions, the 
system determines whether no viewing is occur- 

60 ring, whether a television broadcast is being 
viewed, whether a television broadcast is being 
recorded, whether viewing and recording is 
occurring simultaneously, or whether a pre- 
viously recorded program is being viewed. Once 

65 the viewing mode has been established, the 
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program being viewed is identified by extracting 
a characteristic signature from the video signal 
and storing it for subsequent comparison with 
reference signatures stored in a central office. 

Once the mode of viewing has been deter- 
mined, a signature of the program material being 
viewed or recorded is extracted, provided certain 
predetermined events occur first These events 
shall be referred to as Event 1 and Event 2. 
Several events can serve as Event 1. These events 
may be a turn on of the television set, a channel 
change, a predetermined time interval, for 
example 5 to 10 minutes, since the occurrence of 
the previous Event 1, or a sudden change to a 
black scene. Events that can serve as an Event 2 
include a scene change to a scene other than a 
black scene, a color change, which may be an 
overall color change or a color change of a 
predetermined line, a still picture as evidenced by 
several consecutive substantially identical 
frames, an audio level change, and others. 

Upon the occurrence of an Event 1, the home 
unit monitors the broadcast for the occurrence of 
an Event 2, and extracts a pair of signatures from 
the video signal immediately following the next 
two Event 2s following an Event 1. Or, if a 
previously recorded program is being viewed, the 
system extracts three or more signatures from the 
video signal immediately following the next three 
or more Event 2s following an Event 1. The 
signatures are extracted by detecting and digitiz- 
ing the envelope of the video signal immediately 
following the Event 2s. The digitized signatures, 
as well as the times that the signatures were 
extracted, are stored for future comparison with 
reference signals stored in a central location. The 
time intervals between Event 2s are also stored 
and used to identify the program either by thein- 
selves or in conjunction with extracted sig- 
natures. The time intervals between Event 2s are 
particulariy useful for identifying the playback of 
previously recorded video tapes for reasons that 
will be discussed in a subsequent portion of the 
specification. 

The unit used to acquire the reference sig- 
natures that are compared with the signatures 
received from the home units is similar to the 
home unit except that the reference signature 
acquisition unit monitors each broadcast con- 
tinuously, and does not require an Event 1 to 
initiate a signature extraction. Therefore, each 
broadcast is continuously monitored and a signa- 
ture is extracted each time an Event 2 is detected. 
The signatures, as well as the times of occurrence 
of the signatures are stored to form a library of 
reference signatures. The stored signatures are 
compared with signatures obtained .from the 
home unit that occurred at approximately the 
same time. The time intervals between Event 2s 
are also stored for comparison with corre- 
sponding time intervals received from the home 
units. 

A mode of implementation and the major 
advantages of the present invention will become 
readily apparent upon consideration of the 


following detailed description and attached draw- 
ing wherein: 

RG. 1 is a block diagram of a home unit of the 
program identification system according to the 
5 present invention; 

FIG. 2 is a block diagram of a central office and 
reference signature extraction portions of the 
system according to the invention; 
FIG, 3 is a chart illustrating the various modes • 
10 of viewing and recording that can occur in a 
typical household; 

FIGS. 4 and 5 are flow charts illustrating the 
logical steps performed by the home and central 
office units, respectively; 
IS RG. 6 is a flow chart illustrating the logical 
steps performed by the central office unit in 
identifying signatures; and 

FIG. 7 illustrates how signatures and scene 
changes may be extracted from the video 
20 envelope. 

Referring now to the drawing, with particular 
attention to FIG. 1, there is illustrated a block 
diagram of the home unit of the system according 
to the invention generally designated by the 
25 reference numeral 10. The system 10 receives 
signals to be identified from a television receiver 
12, and in many instances from a home video 
recorder, such as a video cassette recorder or a 
VCR 14, In a typical home installation, the receiver 
so 12 and the VCR 14 receive signal from a receiving 
antenna 16, and in many instances from a cable 
t^evision system 18. These signals may be 
applied directly to the receiver 12 and to the VCR 
14, but it is convenient to employ a switching 
3S network 20 that selectively switches the receiver 
12 and VCR 14 to the antenna 16 and to the cable 
18 in order to permit the viewing and recording of 
broadcasts received from the antenna 16 or from 
the cable 18. In addition, the switching network 20 
40 may connect the VCR 14 to the receiver 12 to 
permit prerecorded or previously recorded tapes 
to be played back by the VCR and viewed by the 
receiver 1 2, 

Video signals representative of the program 

45 applied to the receiver 1 2, as well as video signals 
representative of the signals being recorded by 
the VCR 14, are applied to a video processing 
circuit 22, for example, by a video switching 
circuit 24 which permits either the receiver 12 or 

so the VCR 14, or both, to be monitored. The video 
processing circuit 22 includes a detector and a 
low pass filter (discussed subsequently in greater 
detail), and provides a signal representative of the 
envelope of the video signal to an analog-to- 

55 digital converter 25 which generates a digital 
representation of the video envelope. The 
digitized video envelope signal is applied to an 
events detector 26 which detects predetermined 
events that occur in the digitized video signal, and 

60 causes the signature of the digitized video signal 
to be extracted by signature extraction circuitry 28 
upon the occurrence of a predetermined 
sequence of events. The signature thus generated 
is stored in a data storage system 30 along with 

65 the time that the signature was extracted. The 
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extraction times are provided by a clock 32 which 
provides a digital representation of time to the 
data storage system 30 via, for example, a com- 
bining circuit 34. The clock 32 may be a real time 
clock or a relative time clock that is periodically 
reset from a central location. The function of the 
events detector 26, the signature extractor cir- 
cuitry 28, the data storage 30 and the clock 32 can 
be performed by Individual circuits as shown or 
by a microprocessor based system. 

The home unit 10 Is interrogated at periodic 
intervals, for example, once a day, by a central 
office unit, generally designated by the reference 
numeral 40 (FIG. 2), which compares the sig- 
natures from the various home units with ref- 
erence signatures In order to identify the sig- 
natures from the home unit. The home units may 
be periodically Interrogated by a data communi- 
cations circuit 42 that accesses the various home 
units via a suitable communications system, for 
example, a plurality of telephone lines 44. The 
signatures thus collected are compared by a 
central computer system 48 with signatures 
stored In a data base 46, Central computer system 
48 controfs the collection and classification of the 
signatures received from the home unit as well as 
the generation of reference signatures to be 
stored in the data base 46. The latter function is 
performed in conjunction with a plurality of ref- 
erence signal extraction circuits 50 that are 
located in the cities being monitored. The ref- 
erence signature extraction circuits monitor the 
various networks, cables and other signal sources 
in those cities, and extract reference signals, for 
example, whenever an Event 2 occurs in any of 
the signals being monitored. The reference 
signals thus extracted are transmitted to the 
central office unit 40 and stored in the data base 
46 along with the times that such signatures are 
extracted, as provided by a clock 52. The clock 52 
of FIG. 2 is similar to the clock 32 of the home unit 
and serves to indicate the time of occurrence of 
the reference signatures. The clock 52 rnay also 
be a real time clock or a relative time clock that is 
periodically set by the central office 40. A central 
reference signature extraction circuit 54 and asso- 
ciated clock 56 serve to extract signatures and 
times of extraction of cable originated programs 
and signatures of prerecorded programs. Altema- 
tively, extraction circuits may be placed at the 
head ends of cable system to extract reference 
signatures of cable programs. The signature 
extraction performed by the various extraction 
circuits and data storage is controlled by the 
central computer system 48. 

The home unit of the system according to the 
invention monitors the mode of viewing by 
monitoring the on and off and other functions of 
the television receiver and any video recorder that 
may be used in the home. The various modes of 
viewing or recording are illustrated in FIG. 3. 
Mode 1 occurs when both the VCR and television 
receiver are both off, and indicates that no view- 
ing is taking place. Mode 2 represents television 
viewing and occurs when the television receiver 


is on and the VCR is off. Mode 3 represents 
recording by the home VCR and occurs when the 
television receiver is off and the VCR is on. Mode 
4 occurs when the VCR and the television receiver 

5 are both on, and can represent one of three 
conditions. The first condition, designated as 
mode 4.0, represents television viewing through 
the tuner of the VCR without recording. Mode 4,1 
represents VCR recording and television viewing, 

10 while the third condition, designated as Mode 4.2, 
represents VCR playback. Other modes of oper- 
ation of the VCR, such as fast forward and rewind, 
are designated as Modes 4.3 and 4.4, respectively. 
Most of these modes can be easily detected by 

15 monitoring power line voltage or voltages else- 
where in the television set and the video recorder. 
Modes 4.0, 4.3 and 4.4 can be detected by logic 
circuitry. 

As previously stated, the home unit 10 monitors 

20 the viewing mode and the video signal, and 
extracts signatures from the video signal, only if 
certain events occur. The logic necessary to pro- 
vide this function is illustrated in RG. 4. As is 
apparent from FIG. 4, the mode of operation is 

25 recorded whenever there is a change in the mode, 
such as, for example, a change from viewing to 
recording, etc. Each time the mode changes, a 
determination is made in order to determine what 
the new mode is. This is accomplished by deter- 

30 mining whether the television is on or off, 
whether the VCR is on or off, and whether the VCR 
is in a record or playback mode. Once it has been 
determined whether the television and VCR are 
on or off, and whether the VCR is in a record or 

35 playback mode, the mode can readily be deter- 
mined by using a look-up table containing the 
information in FIG. 3, and the mode is recorded. 
If the mode is determined to be Mode 1, 
indicating no viewing,. no further action is taken 

40 until the mode changes again. If it is determined 
that the mode of operation is either Mode 2 or 
Mode 4, the system monitors the television 
receiver for the occurrence of an Event 1, which 
may, for example, be any one of the following: 

45 1. television turn-on 

2. a channel change 

3. a predetermined time interval (e.g 5—10 
minutes) since the previous Event 1 

4. a sudden scene change to a black scene 

50 5. loss of synchronization for an appreciable 
period of time. 

Such Event Is are relatively easy to detect. For 
example, television turn-on can be detected 
simply by monitoring an appropriate voltage or 

55 current Television turn-on is not used as an Event 
1 in the system illustrated in FIG. 4 but may be 
used in other systems, particularly systems that 
do not monitor a VCR and consequently need not 
employ mode logic, A channel change can be 

60 detected by monitoring the position of the tuner 
mechanism when mechanical tuners are used, by 
monitoring tuning voltage in electrically tuned 
tuners, by monitoring local oscillator frequency, 
or even more simply by monitoring the video 

65 synchronization pulses, preferably the vertical 
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synchronization pulses and indicating a channel 
change in the event of a loss or a change in the 
synchronization pulse for a short time interval. 
The predetermined time interval can readily be 
determined from a clock, and a sudden scene 
change to black can be readily detected by 
monitoring the average value of the video signal 
and indicating a change to a black scene when the 
average value becomes that representative of an 
all black scene. A loss of synchronization for an 
appreciable length of time (longer than that 
caused by a channel change) indicates fast for- 
ward or rewind mode of operation of a VCR 
(Mode 4,3 or 4.4) or the loss of the video signal 
(e.g. station off the air). 

As long as no Event 1 occurs, the home unit 
records no new data; however, upon the occur- 
rence of an Event 1, the system is conditioned to 
extract a signature upon the occurrence of the 
next two consecutive Event 2s, with an Event 2 
being defined as: 

1. a scene change to a scene other than a black 
scene 

2. a color change in successive frames or 
portions thereof 

3. a still picture 

4. an audio level change. 

Ways of detecting the various Event 2s 
described above, as .well as ways for extracting 
signatures will be discussed in a subsequent 
portion of the specification. 

Upon the occurrence of an Event 1, a counter is 
set to zero and the video signal is monitored to 
determine if the television synchronization or 
sync signal is stable. This is accomplished, for 
example, by monitoring the vertical synchroniza- 
tion pulses and waiting until a predetermined 
number, for example, 1 6, consecutive stable verti- 
cal sync pulses are detected before the unit is 
permitted to detect an Event 2. If the synchroniza- 
tion has not been stable for a significant period of 
time, more than five seconds, this period will be 
marked either as television station is off the air or 
VCR fast forward or rewind. 

Once the sync has been stable for the required 
number of pulses, the system monitors the video 
signal for the occurrence of an Event 2. Whenever 
such an Event 2 change is detected, the time of 
the occurrence of the Event 2 is recorded as is the 
time interval since the previous Event 2, if the 
time intervals between Event 2s are to be used as 
a means of program identification. The signature 
of the first frame following the last detected Event 
2 change is then extracted. Since a standard 
broadcast television frame consists of two inter- 
laced fields, and since the two fields that form a 
frame are quite similar, it is not necessary to 
analyze both fields of the frame to obtain a 
signature. Typically, a satisfactory signature Is 
obtained from only a single field, for example, the 
first field of a frame. The method of extracting the 
signature will be discussed in greater detail in a 
subsequent portion of the detailed description, as 
will the method of detecting an Event 2. 

Once the signature has been extracted, the 


counter is incremented by 1. If the count in the 
counter is less than two, indicating that more 
signatures are needed, the system is conditioned 
to respond to another Event 2 to extract and 

5 record another signature. If the count in the 
counter is two (or the required number of sig- 
natures) or greater, indicating that the signatures 
required have been extracted, the system Is con- 
ditioned to require an Event 1 to occur before it 

10 responds to any further Event 2s. 

If it is determined that the mode of operation is 
Mode 3, that is, VCR recording, it is not necessary 
for an Event 1 to occur before the signal is 
monitored for Event 2s. The only determination 

15 that is made is whether the synchronization is 
stable, and if so, the system looks for Event 2s 
continuously and extracts the signature and 
records the time of occurrence of the signature 
each time an Event 2 is detected. Thus, all sig- 

20 natures following every Event 2 are extracted and 
stored so that upon subsequent playback, the 
program may be identified even though only 
portions of the tape may be played back. 
The logic for extracting reference signatures is 

25 illustrated in FIG. 5. Each broadcast and cable 
channel is monitored continuously by the ref- 
erence signature extraction circuits 50 and 54, and 
a signature is extracted each time an Event 2 
occurs to provide a reference library of reference 

30 signals. Consequently, since it is desired to 
extract a signature that occurs after every Event 2, 
Event Is are not monitored. All that is required is 
that the sync be stable and that an Event 2 occur. 
Thus, whenever an Event 2 occurs, the time of 

35 occurrence of the Event 2 as well as the signature 
of the scene following the scene change is 
recorded for future comparison with signatures 
received from the various home units. The time 
intervals between Event 2s may be recorded or 

40 obtained from the times of occurrence of the 
Event 2s. 

The logic employed by the central computer 
system 48 of the central unit 40 in orderto identify 
a program is illustrated In FIG. 6. As is illustrated 
45 In FIG. 6, two consecutive signatures from a home 
unit are selected. If there Is not enough data to 
define two consecutive signatures, the next home 
unit Is monitored. After two consecutive sig- 
natures are selected, a check is made to deter- 
so mine whether the signatures really are a pair, for 
example, that there is no Event 1 between the two 
signatures. 

If the two signatures are a pair, the time 
associated with the first home unit signature is 

55 read. All reference unit signatures in the data base 
46 that occurred within a predetermined time 
Interval of the occurrence of the first home unit 
signature, for example, plus or minus eight 
seconds, are selected for comparison, and the 

60 correlation coefficient between the first home unit 
signature and the selected reference unit sig- 
natures are computed. If one of the correlation 
coefficient exceeds a predetermined threshhold, 
the time associated with the second home unit 

65 signature is read. The next several, for example. 
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six, reference unit signatures that occur, for 
example, within a predetermined time interval, 
for example, plus or minus eight seconds of the 
time associated with the second home unit signa- 
ture are selected, and the correlation coefficients 
between the second home unit signature and the 
reference unit signatures are computed. If one of 
the correlations coefficient exceeds the predeter- 
mined threshold, an identification is made and 
stored, and printed out, if necessary. 

As discussed above, the system according to 
the invention utilizes a time reference to access 
reference signatures that occurred at approxi- 
mately the same time as the home unit signature 
being evaluated in order to eliminate the need for 
comparing the home unit signature with all of the 
reference signatures in the data base 46. How- 
ever, when a previously recorded program is 
being played back, the time that the signature 
occurs during playback cannot be used to locate 
the reference signal. Thus, in accordance with 
another important aspect of the present inven- 
tion, the times of occurrence of signatures that 
occur when programs are being recorded are also 
stored. The times of occurrence of signatures that 
occur in Mode 3 (recording) are kept in^ a separate 
recording time file in the data base 46. Similarly, 
in Mode 4,1 (viewing and recording) the times of 
signatures are also kept in the recording time file. 
Thus, when the broadcast is played back (Mode 
4.2), even though the recorded signatures cannot 
be matched to signatures stored in the reference 
unit during playback, the signature can still be 
identified by utilizing the recording time instead 
of the playbacktimeto retrieve the corresponding 
reference signatures from the reference signature 
data base 46. Thus, by keeping signatures relating 
to programs that were broadcast several days or 
weeks ago within the data base 46, such pre- 
viously recorded and subsequently played back 
broadcasts can be readily identified. 

Alternatively, in order to identify the playback 
of previously recorded programs, instead of com- 
paring the obtained during playback with ref- 
erence the signatures that represent the pro- 
grams broadcast several days or weeks ago 
stored in the recording time file of the reference 
signature data base 46 (FIG. 2) of the central office 
unit the played back signatures may be com-, 
pared with signatures obtained by the home unit 
previously during VCR recording (Mode 3) and 
stored in the data storage 30 (FIG. 1) of the home 
unit. If desired, the signatures may also be sent to 
the central office, as are the signatures obtained 
during normal television viewing, but these sig- 
natures would be labeled as VCR recording, and 
may be stored in the recording time file. However, 
when playback occurs, the played back signatures 
would be compared with the signatures stored in 
the home unit data storage. Only if a match 
occurred, would the signatures be sent to the 
central office and compared with the signatures 
stored in the central office data base 46 to identify 
the program. If no match occurred, the signatures 
would be sent This is because whatever is being 
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played back cannot be identified, and may be 
something like, for example, a home movie. Thus, 
the system eliminates the need to search the 
entire recording file of the data base 46 in an 
5 attempt to identify something that cannot be 
identified. 

In the playback mode, when attempting to 
match the signatures obtained during playback 
with the signatures stored in the data storage 30, 

10 it is convenient to use the time intervals between 
the occurrence of the signatures (or Event 2s( to 
generate a time interval signature, TIS, (discussed 
in greater detail in a subsequent portion of the 
specification) to obtain the match. When using a 

15 time interval signature for matching, three or 
more consecutive signatures are extracted, and 
the time intervals between the signatures are 
compared with the time intervals between sig- 
natures stored in the data storage 30. When a 

20 corresponding pair of consecutive time intervals 
is found, a match is indicated. If greater accuracy 
is desired, more than three consecutive sig- 
natures may be extracted to provide more than 
two consecutive time intervals. 

25 Event 2s may be detected in a variety of ways. 
For example, the video signal may be low pass 
filtered, for example, through a low pass filter 
having a cut-off frequency on the order of 
approximately 2 to 6 kiloHertz, and its average 

30 amplitude detected. The signal may be digitized 
by assigning a number representative of the 
average amplitude of the video signal for each 
frame. The numbers thus obtained may be com- 
pared in various ways. For^example, the numbers 

35 obtained from different frames may be compared, 
and if the numbers are substantially different, a * 
scene change will be indicated. The numbers 
compared may be obtained from consecutive 
frames or from non-consecutive frames, for 

40 example, frames spaced by a 20 to 40 frame 
interval, depending on the type of event that has 
been defined as the Event 2. Alternatively, a 
moving average of the numbers representative of 
the video amplitudes of the last several frames 

45 may be calculated, and the number repre- 
sentative of the average amplitude of the last 
detected frame compared with the moving aver- 
age to indicate a scene change if the number 
associated with the last frame substantially differs 

50 from the moving average. 

The various comparing schemes are useful for 
identifying various different types of Event 2s. For 
example, comparing the number associated with 
a particular frame with the number associated 

55 with a previous frame is useful for detecting an 
Event 2 that may be termed an instant scene 
change. A moving average comparison or a com- 
parison with frames that occurred several frames 
ago is useful for detecting an Event 2 that may be 

60 termed a fade type scene change wherein one 
scene is gradually faded in and the previous 
scene is gradually faded out Because in a fade 
type scene change, the change occurs gradually, 
there is no drastic change in the number asso- 

65 ciated with consecutive frames, and con- 
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sequently, such a scene change cannot readily be 
detected by a system that compares immediately 
successive frames. Numbers that are sub- 
stantially identical for several consecutive frames 
may be used to indicated a still picture. 

Other methods of detecting an Event 2 include 
detecting a color change, for example, by detect- 
ing the average amplitude of a particular color 
component, such as, for example, a red com- 
ponent, of each frame or a particular line or lines 
of each frame, and making comparisons of the 
types described above. Alternatively, different 
color components or an average of the three color 
signals can be monitored for each frame and 
comparisons made. In addition, rather than 
monitoring an entire frame, only portions of a 
frame may be monitored. These portions may be 
selected to be those portions of a frame that 
normally do not change unless a scene change 
occurs. Such portions may occur near the top or 
bottom of the screen because such areas are less 
likely to be affected by action or movement 
occurring within a scene. The Event 2s may be 
detected electronically by monitoring the video 
envelope, or by monitoring the cathode ray tube, 
for example, by placing a light sensitive photo- 
electric device 36 (FIG. 1) in proximity with the 
cathode ray tube, preferably in one corner 
thereof. Alternatively, the audio signal may be 
monitored by an audio processing circuit 38 and 
applied to the analog-to-digital converter 25 to 
generate an Event 2 when the audio signal 
departs from a predetermined level. 

In accordance with an alternative embodiment 
of the present invention that is particulariy suit- 
able for identifying prerecorded programs, the 
time intervals between events, or in changes in 
one or more predetermined parameters, can be 
used to define a feature string. Such a systenn can 
be used either alone, or in combination. with a 
signature extracting system to identify the pro- 
gram. When used alone, the amount of data 
generated by the system is less than that genera- 
ted by a signature generating system, and there- 
fore, does not require much computation time to 
obtain a correlation. Consequently, it is not be 
necessary to store the time of occurrence of each 
change (unless it is desired for other reasons), but 
rather, only the elapsed time between such 
changes need be stored. Such a system is particu- 
lariy useful in identifying the playback of pre- 
recorded programs, or delayed broadcast pro- 
grams because real time data is not necessary to 
make the identification. 

In a system wherein the intervals between 
predetermined changes in predetermined events 
are used to generate a time Interval signature 
(TIS). The time interval signature, which can be 
represented as a digital code sequence, can be 
constructed by measuring the time intervals 
between the sequential occurrences of regularly 
occurring events in a broadcast. Preferably, the 
chosen events should have the following 
properties: 
1. Be readily measurable automatically. 


2. Occur at a sufficiently high rate that enough 
intervals are available to allow unique identifica- 
tion of the signal. 

3. Always be present 

5 4, Cannot be controlled by program originators 
to render identification vulnerable to changes in 
programming methods. 

Typical events that can be used as the events to 
generate a time interval string includes scene 

w changes to black. Event 2s, including shifts in the 
average amplitude of the video signal, departures 
in the amplitude of the instantaneous video signal 
from the average, shifts in the video color bal- 
ance, changes in the average color of a predeter- 

15 mined line, for example, a line near the top of the 
screen, a predetermined audio power level, for 
example, silence, etc. Thus, to define a time 
interval string, the home unit would a generate 
time interval string (TIS) by detecting one of the 

20 predetermined events, for example, an Event 2 
such as a scene change, and store the time 
intervals between consecutive changes. 

The monitoring of events can be done con- 
tinuously to generate a long TIS without requiring 

25 an excessive memory capacity because such 
changes occur relatively infrequently, and the 
time intervals therebetween can be defined by 
only a few bits. The TIS's thus obtained from the 
home unit are compared with reference TIS's 

30 stored in the central unit 

However, in order further to reduce the amount 
of data that must be stored, the time intervals 
between the events being monitored may be 
monitored only after the occurrence of another 

3$ predetermined event, such as an Event 1 ; and it 
has been found that satisfactory identification can 
be obtained by generating a feature string con- 
sisting of three time intervals between changes in 
the monitored variable, such as a scene change or 

40 other Event 2. The time interval strings thus 
generated are matched with reference time inter- 
val strings that are obtained in much the same 
way as those generated by the home units except 
that the reference time interval strings are con- 

45 tinuously generated for each broadcast or other 
program source being monitored. 

Referring now to Fig. 7, there is shown a video 
envelope 100 as provided from the video pro- 
cessing circuit 22 (Fig. 1) to the analog-to-digital 

50 converter 25. The analog-to-digital converter 25 
takes a plurality of samples 102 (Fig. 7) for 
example, 80 samples during a single frame inter- 
val, and generates a digital value representative 
of the amplitude of each of the samples 102. The 

55 digital values thus generated are applied to the 
events detector 26 (Fig. 1) and to the signature 
extraction circuit 28 and used to detect the 
various events and to extract signatures. 
In order to detect an Event 2, for example, a 
60 scene change, the events detector 26 compares 
each of the digitized samples 102 of a current 
frame with a corresponding digitized sample 102 
of a previous frame, either the immediately pre- 
ceding frame or another previous frame, depend- 
65 ing on the type of scene change to be detected. 
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The counterpart samples are then compared to 
determine whether a scene change has occurred. 
More specifically, the detection of a scene change 
occurs as follows: 

1. Take, for example, 80 samples of the video 
envelope of each frame 

2. Compare counterpart samples of a current 
frame and a previous frame 

3. Take the absolute value of the difference of 
each of the counterpart samples 

4. Take the average of the absolute values 

5. Indicate a scene change if the average of the 
absolute values exceeds a predetermined 
threshold. 

The indication of a scene change is applied to 
the signature extraction circuit 28 (Rg. 1) to cause 
the signature extraction circuit 28 to operate on 
the digitized samples 102 to extract the signature. 

The signature extraction circuit 28 operates on 
the digitized samples 102 by taking the average 
value of consecutive samples 102, for example, 
eight consecutive samples to generate a total of, 
for example, ten features 104 that comprise an 
image signature. Typically, four bits define a 
feature, and consequently, an image signature 
can be defined by forty bits of data for the 
example discussed above; however, in some 
instances more features and more data may be 
required. 

Both the home unit 10 (Rg. 1) and the reference 
signature extraction circuits 50 and 54 (Rg. 2) 
operate in the same manner to extract signatures, 
the only difference in operation being that the 
signature extraction circuit 28 (Rg. 1) requires an 
Event 2 to be preceded by an Event 1 before 
signature extraction occurs; but the reference 
signature extraction circuits 50 and 54 (Rg. 2) do 
not require an Event 1 to precede an Event 2, but 
rather extract a reference signature each time an 
Event 2 occurs. 

Once the home unit signatures and the ref- 
erence signatures are obtained, the home unit 
signatures will be compared witK the reference 
unit signatures, normally with those signatures 
occurring in corresponding time intervals. The 
correlation coefficients between the home unit 
signatures and the reference unit signatures will 
be compared, and the reference signature that 
has the best correlation to the home unit signa- 
ture will be used to identify the program being 
viewed, provided that the correlation coefficient 
exceeds a minimum threshold. If it does not, no 
match will be indicated. 

Claims 

1. A system for recording information relating 
to television subscribers' viewing habits by iden- 
tifying viewed programme material selected from 
a plurality of available viewing modes, each mode 
being associated with the status of the television 
receiver and its associated video recorder, in 
which system viewed programmes are identified 
by extracting predetennined characteristics from 
the viewed programme material and recording 


for analysis the said predetermined characteris- 
tics, characterized by means (26) for detecting a 
first predetermined event (Event 1) relating to the 
selected viewing mode, means (26) for detecting 

5 a second predetermined event (Event 2) in the 
viewed programme material, and means (28) for 
extracting at least one predetermined characteris- 
tic of the viewed programme material after the 
occurrence of a predetermined number of said 

10 first and second events. 

2. The system recited in claim 1 wherein said 
program material is that of a television video 
signal and said second event detecting means 
(26) includes means for detecting a scene change 

15 from one video scene to another video scene. 

3. The system recited in claim 2 wherein said 
television signal is received by a television signal 
receiver (12), and wherein said first event detect- 
ing means (26) includes means for detecting the 

20 turn-on of the television signal receiver. 

4. The system recited in claim 2 wherein said 
television signal is received by a television signal 
receiver (12), and wherein said first event detect- 
ing means (26) includes means for detecting a 

25 change in the channel to which the television 
signal receiver is tuned. 

5. The system recited in claim 1 wherein said 
program material is that of a television video 
signal and said first event detecting means (26) 

30 includes means for detecting a change from a 
video scene to a black scene. 

6. The system recited in claim 1 wherein said 
first event detecting means (26) includes means 
for detecting the elapse of a predetermined length 

35 of time after the occurrence of the previous first 
event (Event 1). 

7. The system recited in claim 6 wherein said 
first event detecting means (26) includes means 
for setting the predetermined length of time to 

40 approximately 5 to 10 minutes. 

8. The system recited in claim 1 wherein said 
extracting means (28) includes means for extract- 
ing said predetermined characteristic upon the 
third occurrence of the second event (Event 2) 

45 following the occurrence of the first event (Event 
1), and wherein the predetermined characteristic 
is the time interval between the second event and 
the immediately preceding second event that was 
not followed by a first event. 

so 

Patentanspruche 

1. System zum Aufzeichnen von Informationen 
in bezug auf die Zuschaugewohnheiten von Fern- 

55 sehteilnehmern durch Identifizierung betrachte- 
ten Programmaterials aus einer Mehrzahl verfug- 
barer Betrachtungsmoden, wobei ein jeder 
Modus mit dem Zustand des Fernseh em pf angers 
und seines zugehorigen Video recorders zusam- 

60 menhangt und betrachtete Programme in dem 
System durch Extrahieren vorbestimmter Charak- 
teristika aus dem betrachteten Prog ram mate rial 
und Aufzeichnen der vorbestimmten Charakteri- 
stika zu einer Analyse identifiziert werden, 

65 gekennzeichnet durch Mittel (26) zur Ermittlung 
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eines ersten vorbestimmten Ereignisses (Ereignis 
1), das sich auf den gewahlten Betrachtungsmo- 
dus bezieht, Mittel (26) rur Ermittlung eines zwei- 
ten vorbestimmten Ereignisses (Ereignis 2) in 
dem betrachteten Programmaterial und Mittel 
(28) 2um Extrahieren von mindestens einem vor- 
bestimmten Charakteristikum des betrachteten 
Programmaterials nach dem Auftreten einer vor- 
bestimmten Anzahl der genannten ersten und 
zweiten Ereignisse. 

2. System nach Anspruch 1, worin das Pro- 
grammaterial dasjenige eines Femseh-Video- 
signals ist und die Mittel (26) zur Ermittlung des 
zweiten Ereignisses Mittel zur Ermittlung eines 
Szenenwechsels von einer Videoszene zu einer 
anderen enthalten. 

3. System nach Anspruch 2, worin das Fernseh- 
signal von einem Femsehsignalempf anger (12) 
empfangen wird und die Mittel (26) zur Ermittlung 
des ersten Ereignisses Mittel zur Ermittlung des 
Einschaltens des Fernsehsignalempfangers ent- 
halten. 

4. System nach Anspruch 2, worin das Femseh- 
signai von einem Fernsehsignalempfanger (12) 
empfangen wird und die Mittel (26) zur Ermittlung 
des ersten Ereignisses Mittel zur Ermittlung eines 
Wechsels in dem Kanal enthalten, auf den der 
Fernsehsignalempfanger eingestellt ist. 

5. System nach Anspruch 1, worin das Pro- 
grammaterial dasjenige eines Fernseh-Video- 
signals Ist und die Mittel (26) zur Ermittlung des 
ersten Ereignisses Mittel enthalten fur die Fest- 
stellung eines Wechsels von einer Videoszene zu 
einer schwarzen Szene. 

6. System nach Anspruch 1, worin die Mittel 
(26) zur Ermittlung des ersten Ereignisses Mittel 
enthalten fur die Feststellung des Ablaufs einer 
vorbestimmten Zeit seit Auftreten eines voraus- 
gehenden ersten Ereignisses (Ereignis 1). 

7. System nach Anspruch 6, worin die Mittel 
(26) zur Ermittlung des ersten Ereignisses Mittel 
enthalten zur Festsetzung der vorbestimmten Zeit 
auf etwa 5 bis 10 Minuten. 

8. System nach Anspruch 1, worin die Extrak- 
tionsmitte! (28) Mittel enthalten zum Extrahieren 
des vorbestimmten Charakteristikums nach dem 
dritten Auftreten des zweiten Ereignisses (Ereig- 
nis 2) im AnschluS an das Auftreten des ersten 
Ereignisses (Ereignis 1) und worin das vorbe- 
stimmte Charakteristikum das Zeitintervall 
zwischen dem zweiten Ereignis und dem unmit- 
telbar vorausgehenden zweiten Ereignis, nach 
dem kein erstes Ereignis auftrat ist. 

Revendications 

1. Systeme pour I'enregistrement d'informa- 
tions relatives aux habitudes de vision des 
abonnes i la television en identifiant le contenu 
d'un programme visionne selectionn6 a partir 
d'une pluralite de modes de vision disponibies, 
chaque mode 6tant associ6 au statut du r6cepteur 
de television et de I'enregistreur vid§o qui lui est 
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associe, les programmes visionn6s dans ledit 
systeme etant identifies en extrayant des caracte- 
ristiques predeterminees du contenu du pro- 
gramme visionne et en enregistrant, en vue de 

5 leur analyse, lesdites caracteristiques predetermi- 
nees, caracterise par des moyens (26) pour detec- 
ter un premier evenement predetermine (Evene- 
ment 1) relatif au mode de vision seiectionne, des 
moyens (26) pour detecter un second 6v6nement 

10 predetermine (Evenement 2) dans le contenu du 
programme visionne, et des moyens (28) pour 
extraire au moins une caracteristique predeternii- 
nee du contenu du programme visionne apres 
que se soient produits un nombre predetermine 

75 desdits premier et second ev6nements. 

2. Systeme suivant la revendication 1, dans 
lequel ledit contenu du programme est celui d'un 
signal video de television et ledit moyen de 
detection dudit second evenement (26) comporte 

20 des moyens pour detecter un changement de 
scene d'une scene vid6o h une autre. 

3. Systeme suivant la revendication 2, dans 
lequel ledit signal de television est regu par un 
recepteur de signal de television (12) et oCi le 

25 moyen de detection (26) dudit premier evene- 
ment comporte un moyen pour detecter la mise 
en service du recepteur de signal de television. 

4. Systeme suivant la revendication 2, dans 
lequel ledit signal de television est regu par un 

30 recepteur de signal de television (12) et oCi le 
moyen de detection (26) dudit premier evene- 
ment comporte un moyen pour detecter un 
changement du canal sur lequel est accorde le 
recepteur du signal de television. 

35 5. Systeme suivant la revendication 1, dans 
lequel ledit contenu du programme est celui d'un 
signal vid6o de television et le moyen de detec- 
tion (26) dudit premier evenement comporte un 
moyen pour detecter un passage d'une scdne 

40 video e une scene noire. 

6. Systeme suivant la revendication 1, dans 
lequel le moyen de detection (26) dudit premier 
evenement comporte up moyen pour detecter la 
duree d'un temps predetermine apres que se soit 

45 produit le premier evenement precedent (Evene- 
ment 1). 

7. Systeme suivant la revendication 6, dans 
lequel le moyen de detection (26) dudit premier 
evenement comporte un moyen pour fixer la 

50 dur6e de temps pred6terminee k environ 5 e 10 
minutes. 

8. Systeme suivant la revendication 1, dans 
lequel ledit moyen d'extraction (28) comporte un 
moyen pour extraire ladite caracteristique prede- 

55 terminee lorsque se produit pour la troisieme fois 
le second evenement (Evenement 2) apr^s que se 
soit produit le premier evenement (Evenement 1) 
et oCi la caracteristique predeterminee est I'inter- 
valle de temps qui separe le second evenement et 

60 le second evenement qui le precede immediate- 
ment et qui n'a pas ete suivi par un premier 
evenement. 
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